Sensitive Talking Heads

Spoken language user interface can dramatically speed up computer use. Unfortunately, if the speech user interface gets in the way too often, the user turns it off. Users are unforgiving: a technology that impairs productivity just once may never get a second chance. In order to give the user interface a fighting chance, why not give it a certain amount of emotional sensitivity? Users respond better to an avatar that displays appropriate emotional nuance; conversely, if the avatar detects extreme frustration on the part of the user, it can hide in the corner of the monitor until the frustration has passed. A hidden avatar is still present, and can continue to be of service to the user, upon request. Audio-only speech synthesis and recognition are now sufficiently accurate to be the foundation for a host of application technologies (see, e.g., the May 2008 issue of Signal Processing Magazine). Automatic recognition and synthesis of emotionally nuanced speech, on the other hand, are still topics of active research. This column describes experiments in emotive spoken language user interface. We find that both recognition accuracy and synthesis quality are improved when one takes advantage of multimodal information, synthesizing and recognizing information in both the audio and video modalities.